{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a quick notebook to demonstrate the pyroSAR functionality for importing processed SAR scenes into an Open Data Cube"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyroSAR.datacube_util import Product, Dataset\n",
    "from pyroSAR.ancillary import groupby, find_datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a directory containing processed SAR scenes\n",
    "dir = '/path/to/some/data'\n",
    "\n",
    "# define a name for the product YML; this is used for creating a new product in the datacube\n",
    "yml_product = './product_def.yml'\n",
    "\n",
    "# define a directory for storing the indexing YMLs; these are used to index the dataset in the datacube\n",
    "yml_index_outdir = './yml_indexing'\n",
    "\n",
    "# define a name for the ingestion YML; this is used to ingest the indexed datasets into the datacube\n",
    "yml_ingest = './ingestion.yml'\n",
    "\n",
    "# product description\n",
    "product_name_indexed = 'S1_GRD_index'\n",
    "product_name_ingested = 'S1_GRD_ingest'\n",
    "product_type = 'gamma0'\n",
    "description = 'this is just some test'\n",
    "\n",
    "# define the units of the dataset measurements (i.e. polarizations)\n",
    "units = 'backscatter'\n",
    "# alternatively this could be a dictionary:\n",
    "# units = {'VV': 'backscatter VV', 'VH': 'backscatter VH'}\n",
    "\n",
    "ingest_location = './ingest'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find pyroSAR files by metadata attributes\n",
    "files = find_datasets(dir, recursive=True, sensor=('S1A', 'S1B'), acquisition_mode='IW')\n",
    "\n",
    "# group the found files by their file basenames\n",
    "# files with the same basename are considered to belong to the same dataset\n",
    "grouped = groupby(files, 'outname_base')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(files))\n",
    "print(len(grouped))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next step we create a new product, add the grouped datasets to it and create YML files for indexing the datasets in the cube."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new product and add the collected datasets to it\n",
    "# alternatively, an existing product can be used by providing the corresponding product YML file\n",
    "with Product(name=product_name_indexed,\n",
    "             product_type=product_type,\n",
    "             description=description) as prod:\n",
    "\n",
    "    for dataset in grouped:\n",
    "        with Dataset(dataset, units=units) as ds:\n",
    "\n",
    "            # add the datasets to the product\n",
    "            # this will generalize the metadata from those datasets to measurement descriptions,\n",
    "            # which define the product definition\n",
    "            prod.add(ds)\n",
    "\n",
    "            # parse datacube indexing YMLs from product and dataset metadata\n",
    "            prod.export_indexing_yml(ds, yml_index_outdir)\n",
    "\n",
    "    # write the product YML\n",
    "    prod.write(yml_product)\n",
    "    \n",
    "    # print the product metadata, which is written to the product YML\n",
    "    print(prod)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a YML file for creating a new product and individual YML files for indexing the datasets, we can create a last YML file, which will ingest the indexed datasets into the cube. For this a new product is created and the files are converted to NetCDF, which are optimised for useage in the cube. The location of those NetCDF files also needs to be defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Product(yml_product) as prod:\n",
    "    prod.export_ingestion_yml(yml_ingest, product_name_ingested, ingest_location, \n",
    "                              chunking={'x': 512, 'y': 512, 'time': 1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cubeenv",
   "language": "python",
   "name": "cubeenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
